Monitoring and Security

All monitoring and security tooling runs on Secops, away from the servers it watches.

Tool	Where to find it	Purpose
Grafana	corais.abair.ie	Dashboards for metrics and logs
Loki	behind corais.abair.ie/loki/	Log storage
Prometheus	behind corais.abair.ie/prometheus/	Metric storage
Uptime Kuma	status.abair.ie	Uptime checks and the public status page
Wazuh	SSH tunnel only	SIEM: security events, vulnerability detection, file integrity
Rocket.Chat	alerts.abair.ie	Team chat and alerts

Metrics and logs

Each server runs a Grafana Alloy agent that pushes data to Secops:

• logs go to https://corais.abair.ie/loki/api/v1/push
• metrics go to https://corais.abair.ie/prometheus/api/v1/write

Both endpoints need HTTP basic auth: user admin, with the password Webserver htpasswd admin. Prometheus scrapes nothing itself; everything arrives by push.

Server	Logs	Metrics
webserver	nginx access log (JSON), labelled with vhost and status	host
api-gateway	/opt/api-gateway/logs/gateway.log	host, plus /metrics from api-gateway and api-authentication
services-user, services-staff, synthesis-legacy	all container logs (from the Docker socket)	host
Amergin	all container logs	host, GPU (DCGM exporter), asr-diarisation /metrics
Puca	the whole systemd journal, including the recording checks	host

Container log lines are labelled with container, instance and host. If a line is JSON with a level field, level becomes a label too.

Docker-build, Secops and Vaultwarden don't run Alloy, so they have no metrics or logs in Grafana.

The Alloy configs are in ansible/templates/alloy/, except webserver and api-gateway, which are written inline in their playbooks, and Puca, which has its own in ansible/puca/files/alloy/. Alloy is redeployed on every setup or update run of the server's playbook.

Retention

Loki has retention turned on but no retention period set, so it keeps logs indefinitely. Prometheus uses its default retention. Both store data in Docker volumes on Secops.

Grafana

Dashboards

The dashboards are JSON files in grafana-dashboards/ in the repo. make update-observability-secops loads every file into Grafana, through Grafana's v2 dashboard API, and puts each in a folder:

Folder	Dashboards	Who can see it
Amergin-server	Amergin, Streaming ASR, Fotheidil, Batch Recognition, Synthesis	admins only
Services	Synthesis (synthesis-services.json), Services User, Services Staff, Webserver	all Grafana users
API Gateway	API Gateway Main, API Gateway Stats	all Grafana users
Púca	Púca - Radio Archiver, Púca - TV Archiver	admins only

To change a dashboard for good, edit the JSON in the repo and re-run the target. Changes made in the Grafana UI are overwritten on the next run. synthesis.json and synthesis-services.json are the same dashboard in two folders, so keep them in step.

The Púca dashboards were originally produced by grafana-dashboards/_gen_puca.py and _gen_puca_tv.py. Those scripts are out of date: they output the old dashboard format with old data-source IDs. Edit the JSON directly instead of regenerating it.

Users and access

• Grafana accounts are created by hand; the playbook never creates users.
• On each run, the playbook reads the Vaultwarden note users.json. Everyone with the role developer who already has a Grafana account is added to the app-developers team and given the org role Editor.
• Folder permissions don't restrict the underlying data. Editors can use Explore to query all of Loki and Prometheus, including data from the "admins only" folders.

Uptime Kuma

Uptime Kuma runs at status.abair.ie, and its public status page is /status/main. Its monitors are set up in its own UI, not in the repo. They check API services both directly and through the gateway, and web apps through their public URLs. Its data was carried over by hand from the old observability VM.

Admins manage monitors from the admin panel at status.abair.ie/dashboard. The root URL doesn't lead to it, so go to /dashboard directly and log in there.

Cloudflare's Bot Fight Mode must stay off for abair.ie. If it's on, Cloudflare blocks Uptime Kuma's checks and every monitor reports a failure.

GPU watchdog

On Amergin, the gpu-watchdog container checks the health of the GPU services every minute and restarts a service after three failed checks in a row. Its JSON log lines (watchdog_restart, watchdog_recovered, and so on) end up in Loki with the other container logs. See Amergin.

Wazuh (SIEM)

The Wazuh manager on Secops collects security events from an agent on each host. It also runs vulnerability detection, file-integrity monitoring (every 12 hours), configuration assessment (every 12 hours) and system inventory (every hour). Email alerts are off.

Agents are installed by ansible/wazuh-agent-tasks.yml. Each host's playbook imports it under the wazuh-agent tag, and each host has its own target (see the Makefile Reference). Each agent:

• installs wazuh-agent 4.14.6 from Wazuh's apt repository;
• enrolls with the manager at 135.181.120.84 on port 1515, using Wazuh Agent Registration Password;
• is named after its inventory host name (webserver-vm, amergin, and so on).

Enrollment is skipped if the agent already has a key. The Secops agent reports to the manager on 127.0.0.1.

Agents exist for: webserver, api-gateway, services-user, services-staff, synthesis-legacy, docker-build, the Proxmox host, Amergin, Secops and Vaultwarden. As of 2026-09-16, scealai was still pending: its target (enroll-wazuh-scealai) exists, but the VM wasn't accepting SSH at the time.

A new server must also be allowed through Secops's firewall on ports 1514 and 1515. Add it to extra_allow_rules in setup-secops-host.yml, and to the allow list in ansible/secops/files/nginx/wazuh-stream.conf. VMs on hetzner-main are already covered, because they all leave through 136.243.40.234.

To open the dashboard, see Secops → Wazuh.